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PREFACE 


This  Memorandum  is  a  part  of  continuing  RAND  studies  on  methods  for 
-Improving  the  reliability  of  Air  Force  systems.  Early  est^^ation  of  reli 
ability  on  the  basis  of  a  few  trials  is  of  value  in  that  it  focuses  atten 
tion  on  critical  reliability  problems.  This  Memorandum  provides  a  new 
approach  to  the  problem  of  reliability  estimation. 
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SUMMARY 


This  Memorandum  presents  a  new  approach  to  the  estimation  of  reliability 
on  the  basis  of  a  few  trials.  In  this  new  method,  based  on  information  theory 
concepts,  when  the  probability  of  success  is  not  knovm,  the  proposed  way  to 
assign  pirobabilities  is  to  maximize  the  measure  of  uncertainty.  This  point- 
estimation  method  is  claaracterized  by  less-extreme  fluctuations  in  estimates 
as  the  results  of  Bernoulli  trials  become  known. 
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I.  imOKJCTION 

The  process  of  testing  to  determine  reliability  is  generally  expensive 
and  time-consuming.  Under  the  pressures  of  costs  and  schedules,  engineers 
are  in  many  instances  required  to  make  estimates  of  reliability  based  on 
the  results  of  tests  of  very  few  units.  In  the  case  of  Bernoulli  trials 
which  have  an  outcome  of  either  success  or  failure,  the  reliability  is 
usually  estimated  as  the  number  of  successes  divided  by  the  total  number 
of  trials.  Very  little  confidence  can  be  placed  in  this  method  of  estima¬ 
tion  if  the  number  of  trials  is  small.  Consider  an  extreme  example;  if  a 
conventional  coin  has  been  tossed  only  once,  the  result  yields  an  estimate 
for  obtaining  heads  as  either  1  or  0.  These  values  are  as  far  removed  from 
the  a  priori  value  of  0.5  as  it  is  possible  to  be  on  the  probability  scale. 
This  type  of  experiment  is,  therefore,  quite  meaningless. 

U'hen  possible  outcomes  of  a  test  are  success  and  failure,  the  familiar 
binomial  distribution  is  applied  to  the  solution  of  many  problems.  The 
equation  for  this  distribution  is 
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where 

p  =  probability  of  success 
q  =  1  -  p  =  probability  of  failure 
N  =  number  of  triads 
m  =  number  of  successes 
n  =  N  -  m  =  number  of  failures 


1  t 

+  q  =  1 


Each  term  of  the  expansion  represents  the  probability  of  obtaining  exactly 
m  successes  and  n  failures  in  N  trials.  It  will  be  observed  that  the  value 
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of  p  must  be  assigned  or  estimated  on  the  basis  of  previous  knowledge  of 
the  characteristics  of  the  population,  or  estimating  as  Many  useful 
applications  of  this  distribution  exist  when  a  priori  knowledge  yields  the 
exact  value  of  p,  such  as  problems  which  are  analogous  to  coin  or  die  tos¬ 
sing  or  drawing  cards  from  a  pack.  Difficulties  in  applications  are  in 
the  estimation  of  p  on  the  basis  of  a  very  few  trials  if  previous  knowledge 
of  p  is  nonexistent.  This  problem  has  been  addressed  in  the  past  by  many 
mathematicians . 

A  brief  summary  of  the  past  work  on  the  problem  is  included  in  a  paper 
by  Steinhaus.^^^  Generally,  it  is  assumed  that  there  is  an  a  priori  proba¬ 
bility  distribution,  f(p),  for  the  unknown  parameter  p.  After  taking  a 
sample,  x,  we  then  have  an  a  posteriori  estimate  of  g(p;x)  for  p.  The 
value  of  p  is  then  assigned  by  applyi.ng  one  of  several  criteria,  such  as 
maximizing  g,  the  mean  value  of  g(pjx)  dp,  or  some  other  function,  depend¬ 
ing  on  whether  we  want  the  "most  probable"  value  of  p,  the  value  that  mini¬ 
mizes  the  mean  square  error,  or  a  value  that  satisfies  some  other  criterion. 
Baye's  original  tentative  suggestion  was  to  take  a  uniform  a  priori  distri¬ 
bution,  i.e.,  f(p)  =  1,  if  one  has  no  initial  knowledge.  More  recently 
the  work  of  Wald  has  suggested  that  other  choices  of  f  may  be  appropriate 
in  various  circumstances,  Steinhaus  discusses  the  situation  when  one 
assumes  that  there  is  a  loss  function  proportional  to  the  mean  square 
error,  and  the  loss  function  is  to  be  minimized. 

This  Memorandum  provides  an  approach  to  the  problem  based  on  infor¬ 


mation-theory  concepts. 
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II .  HEURISTIC  DEVELOPMENT 

An  approach  to  the  problem  of  estimating  success  probabilities  on  the 
knowledge  of  the  outcome  of  a  few  trials  can  be  based  on  the  concepts  of 
information  theo2ry  developed  by  E.  C.  Shannon  in  Ref.  2,  which  is  summarized 
briefly  in  Chapter  3  of  Ref.  3*  These  concepts  lead  to  the  equation 


H  =  -Z  P.  In  P, 
1  i 

1 


where 

H  =  measure  of  uncertainty 

*bll 

P^  =  probability  of  the  i  state 

"til 

In  P^  =  a  measure  of  the  uncertainty  in  the  i  state 


If  we  are  concerned  only  with  the  future  events  of  success  and  failure, 
there  are  only  two  possible  states,  and  this  equation  reduces  to 


H  =  -plnp-qlnq 


If  we  have  conducted  a  number  of  trials  in  which  the  outcome  has  been 
determined,  the  probability  of  m  successes  and  n  failures  in  N  trials  is 
given  by  the  usual  binomial  distribution  result  for  Bernoulli  trials 
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The  logarithm  of  P(ni,  n)  is  a  measure  of  the  information  that  has  been  ob¬ 
tained  from  a  given  number  of  trials,  and  since  the  information  is  in  hand, 
the  probability  of  obtaining  it  is  1.  This  reduces  the  uncertainty,  so  that 
we  can  write  as  a  measure  of  the  residual  uncertainty 


H  =  “p  In  p  -  q  In  q  +  l(  In 
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Here  again  if  p  is  known  from  previous  information,  we  have  a  straightfor¬ 
ward  calculation  to  determine  the  measiare  of  the  residual  uncertainty  H. 

If  p  is  not  known,  the  proposed  way  to  assign  the  probabilities  is  to 
maximize  the  function  H.  This  maximizing  principle,  which  can  only  be 
accepted  as  an  axiom,  has  been  used  in  other  fields  in  which  information 
theory  is  applied.  In  statistical  mechanics,  where  the  quantity  analogous 
to  the  measure  of  uncertainty  is  the  entropy,  Jaynes^^'^^  and  Tribus^^^ 
have  successfully  used  the  maximizing  principle  in  the  solution  of  problems. 
For  example.  Tribus  obtains  the  probability  distribution  of  the  energy 
states  of  a  thermodynamic  system  by  considering  these  states  as  the  possi¬ 
ble  outcomes  of  the  system,  with  unknown  probabilities  of  occurrence,  and 
then  maximizing  the  enti’opy  of  the  dirtribution  subject  to  the  condition 
that  the  expected  value  of  the  energy  of  the  system  is  known.  The  distri¬ 
bution  thus  obtained  is  that  of  the  well-known  "microcanonical  ensemble" 
of  statistical  mechanics.  This  distribution  is  obtained  more  conventionally 
by  a  lengthy  and  difficult  train  of  reasoning  which  is  eliminated  by  the 
maximum -entropy  principle. 

If  H  is  maximized,  we  have 


■Mn  £  =  H 

q  p 


n 
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since  dp  =  -dq. 

If  the  number  of  trials  is  zero,  and  we  are  in  a  state  of  complete 
ignorance  with  regard  to  the  probability  of  success,  we  have 

In  £  =  0 

q 

and  since 


q  =  1  -  p 
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then 


This  yields  p  =  ^  on  solution;  that  is,  if  we  are  in  a  state  of  no  know- 
ledge  with  regard  to  the  probability  of  success,  we  must  regard  success  or 
failure  as  equally  likely . 

In  accordance  kith  this  maximizing  principle,  Fig.  1  and  Table  1  have 
been  prepared  as  grapohlc  solutions  of  the  measure-of -information  equation. 
Figure  1  has  as  abscissa  the  total  number  of  trials,  N,  which  is  to  be  read 
at  integral  values  only.  The  curves  are  symmetrical  about  the  probability 
ordinate  of  0.5.  The  left  ordinate  gives  the  probability  of  success  (re¬ 
liability)  for  a  glY«n  curve  representing  m  successes  and  N  -  m  failures; 
the  right  ordinate  gives  the  probability  of  failure  (lonre liability)  for  n 
failures  and  N  -  n  successes.  Table  1  is  the  numerical  equivalent  of  the 
graphs  of  Fig.  1. 

When  success  prnhability  is  computed  as  the  number  of  successes  di¬ 
vided  by  the  number  nf  trials,  the  probability  oscillates  through  extremes 
in  the  beginning  and  then  converges  slowly  to  the  value  for  a  large  number 
of  trials.  The  valuss  presented  in  Fig.  1  and  Table  1  start  at  the  middle 
of  the  scale  for  zem  trials,  and  as  the  results  of  more  trials  become 
available  the  oscilLstion  is  less  extreme  and  the  computed  probability 
converges  somewhat  nmre  rapidly  to  the  value  for  a  large  number  of  trials. 
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Fig.  i — Probability  assignment  in  Bernoulli  trials 
(after  having  observed  m  successes  and  n  failures  previously) 
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III.  EXAMPLES 

Figure  2  presents  gi’aphically  the  reliability  results,  based  on  Ref. 

7,  of  the  first  15  Atlas  missile  firings.  The  solid  curve  is  computed  by 
the  method  of  maximizing  the  uncertainty.  The  dashed  curve  is  based  on 
the  conventional  method  of  number  of  successes  divided,  by  number  of  trials. 
In  the  beginning  the  solid  curve  oscillates  less  violently  than  the  dashed 
curve.  As  the  number  of  trials  grows  larger,  the  curves  approach  each 
other.  The  relative  initial  stability  of  the  solid  curve  will  lead  to  a 
reliability  less  subject  to  rapid  change  as  successive  initial  firings 
occur. 

Figure  3  is  a  plot,  similar  to  Fig.  2,  obtained  by  using  random  numbers 
to  simulate  repeated  tossing  of  a  perfect  coin.  Even  numbers  were  assigned 
to  "success"  and  odd  numbers  to  "failure."  The  solid  curve  again  shows  less 
oscillation  initially  than  the  dashed  curve.  The  two  curves  approach  each 


other  as  the  nimiber  of  trials  increases 


Fig. 2  —  Reliability  results  of  Atlas  firings 
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IV.  CONCLUSIONS 

The  new  method  of  point  estimation  of  probahilities  for  success  is 
characterized  by  less-extreme  fluctuations  as  the  results  of  Bernoulli 
trials  become  known.  The  estimates  approach  the  conventional  estimate  of 
the  ratio  of  successes  to  total  trials  as  the  munber  of  trials  becomes 
large.'  Application  of  the  method  to  reliability  testing  should  be  made 
to  determine  if  predictions  of  reliability  can  be  made  with  more  assurance 
for  a  given  number  of  units  tested  when  sample  sizes  are  very  small. 

Further  areas  of  interest  include  application  of  the  method  to  prob¬ 
lems  where  there  is  some  knowledge  of  reliability  before  testing,  and 
development  of  interval  estimates  for  various  confidence  levels  based  on 
the  binomial  distribution.  Comparison  of  the  properties  of  this  distri¬ 
bution  with  those  suggested  by  Steinhaus^^^  should  also  be  instructive. 
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